{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for RentalListingInquiries Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第3步：调整树的参数：subsample & colsample_bytree\n",
    "\n",
    "\n",
    "首先import必要的模块"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# path to wehre the data lies\n",
    "dpath = \"D:/AI/week3data_RentalListingInquiries/\"\n",
    "data_train = pd.read_csv(dpath+\"RentListingInquries_FE_train.csv\")\n",
    "\n",
    "data_test = pd.read_csv(dpath+\"RentListingInquries_FE_test.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interest Level分布，看看各类样本分布是否均衡"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#sns.countplot(data_train.interest_level)\n",
    "#pyplot.xlabel('interest_level')\n",
    "#pyplot.ylabel('Number of occurences')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "每类样本分布不是很均匀，所以交叉验证时可以考虑各类样本按比例抽取\n",
    "\n",
    "# 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = data_train['interest_level']\n",
    "X_train = data_train.drop('interest_level',axis = 1)\n",
    "X_test = data_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "各类样本不均衡，交叉验证是采用StratifiedKFlod，在每次采样是各类样本按比例采样"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5,shuffle=True,random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第一轮参数调整得到的n_estimators最优值为220，其余参数继续默认值。max_depth & min_child_weight没训练，n_estimators也就没做第二轮训练。\n",
    "\n",
    "用交叉验证评价模型性能时，用scoring参数定义评价指标。评价指标是越高越好，因此用一些损失函数当评价指标时，需要再加负号，如neg_log_loss，neg_mean_squared_error详见sklearn文档"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'colsample_bytree': [0.7, 0.8], 'subsample': [0.3, 0.4]}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subsample = [i/10.0 for i in range(3,5)] # 受限于机器性能，只选两个值调优\n",
    "colsample_bytree = [i/10.0 for i in range(7,9)] # 受限于机器性能，只选两个值调优\n",
    "param_test3_1 = dict(subsample=subsample,colsample_bytree=colsample_bytree)\n",
    "param_test3_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\model_selection\\_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.58914, std: 0.00342, params: {'subsample': 0.3, 'colsample_bytree': 0.7},\n",
       "  mean: -0.58665, std: 0.00332, params: {'subsample': 0.4, 'colsample_bytree': 0.7},\n",
       "  mean: -0.58864, std: 0.00370, params: {'subsample': 0.3, 'colsample_bytree': 0.8},\n",
       "  mean: -0.58770, std: 0.00403, params: {'subsample': 0.4, 'colsample_bytree': 0.8}],\n",
       " {'colsample_bytree': 0.7, 'subsample': 0.4},\n",
       " -0.5866507899064141)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb3_1 = XGBClassifier(\n",
    "    learning_rate=0.1,\n",
    "    n_estimators=220, # 第一轮参数调整得到的n_estimators最优值\n",
    "    max_depth=5,\n",
    "    min_child_weight=1,\n",
    "    gamma=0,\n",
    "    subsample=0.3,\n",
    "    colsample_bytree=0.8,\n",
    "    colsample_bylevel=0.7,\n",
    "    objective='multi:softprob',\n",
    "    reg_alpha=1.5,  # 在第2步：调整正则化参数：reg_alpha 和 reg_lambda \n",
    "    reg_lambda=1, # Best: -0.588640 using {'reg_alpha': 1.5, 'reg_lambda': 1}\n",
    "    seed=3)\n",
    "\n",
    "gsearch3_1 = GridSearchCV(xgb3_1,param_grid=param_test3_1,scoring='neg_log_loss',n_jobs=-1,cv=kfold)\n",
    "gsearch3_1.fit(X_train,y_train)\n",
    "\n",
    "gsearch3_1.grid_scores_, gsearch3_1.best_params_, gsearch3_1.best_score_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([120.91780005, 145.26660004, 136.90719995, 159.54419999]),\n",
       " 'mean_score_time': array([0.54459991, 0.56059999, 0.54880004, 0.56940002]),\n",
       " 'mean_test_score': array([-0.58913731, -0.58665079, -0.58863999, -0.58769548]),\n",
       " 'mean_train_score': array([-0.52082965, -0.51704625, -0.51831375, -0.51477038]),\n",
       " 'param_colsample_bytree': masked_array(data=[0.7, 0.7, 0.8, 0.8],\n",
       "              mask=[False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_subsample': masked_array(data=[0.3, 0.4, 0.3, 0.4],\n",
       "              mask=[False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'colsample_bytree': 0.7, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.4}],\n",
       " 'rank_test_score': array([4, 1, 3, 2]),\n",
       " 'split0_test_score': array([-0.5840038 , -0.5808993 , -0.58241262, -0.58036951]),\n",
       " 'split0_train_score': array([-0.52196787, -0.51931153, -0.52009783, -0.51751626]),\n",
       " 'split1_test_score': array([-0.5881086 , -0.58547817, -0.58704075, -0.58713972]),\n",
       " 'split1_train_score': array([-0.52085217, -0.51694861, -0.51877524, -0.51488264]),\n",
       " 'split2_test_score': array([-0.58771824, -0.58721656, -0.58912327, -0.58848601]),\n",
       " 'split2_train_score': array([-0.5218292 , -0.51784762, -0.51811925, -0.51460639]),\n",
       " 'split3_test_score': array([-0.59256436, -0.58972635, -0.59223708, -0.59039421]),\n",
       " 'split3_train_score': array([-0.52100227, -0.51660569, -0.51818762, -0.51381885]),\n",
       " 'split4_test_score': array([-0.5932928 , -0.58993457, -0.59238737, -0.59208928]),\n",
       " 'split4_train_score': array([-0.51849673, -0.5145178 , -0.51638881, -0.51302774]),\n",
       " 'std_fit_time': array([1.48650604, 2.36664829, 1.42016563, 2.21526214]),\n",
       " 'std_score_time': array([0.01547387, 0.01344021, 0.00448996, 0.08198202]),\n",
       " 'std_test_score': array([0.00341866, 0.00331714, 0.00370279, 0.00402967]),\n",
       " 'std_train_score': array([0.00124639, 0.00157292, 0.00119625, 0.00151799])}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch3_1.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.586651 using {'subsample': 0.4, 'colsample_bytree': 0.7}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAELCAYAAADkyZC4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xd4VGX2wPHvSafXUAOE3nuooS1KVUFAQV1dsbHqIgprQV0VRRdEfwgqrr2hK7pgAZQqSAkgBAlJAIEACQmhhE6A9Pf3x72QIQQIyUxmkpzP88zDzG1zXkoO7y3niDEGpZRSytm83B2AUkqp4kkTjFJKKZfQBKOUUsolNMEopZRyCU0wSimlXEITjFJKKZfQBKOUUsolNMEopZRyCU0wSimlXMLH3QG4U9WqVU1wcLC7w1BKqSJl8+bNR40xgdfarkQnmODgYMLDw90dhlJKFSkiEpeX7fQUmVJKKZfQBKOUUsolNMEopZRyiRJ9DUYpdan09HQSEhJISUlxdyjKAwQEBBAUFISvr2++9tcEo5S6KCEhgXLlyhEcHIyIuDsc5UbGGI4dO0ZCQgL169fP1zH0FJlS6qKUlBSqVKmiyUUhIlSpUqVAs1lNMEqpS2hyURcU9O+CJph8SDx5ni/Xx5KVpe2mlVLqSjTB5MN34fG8+NM2Rn6wnpgjye4ORymlPJImmHx4/IbG/N/tbdl9JJnBM9cwa2UM6ZlZ7g5LqSLv5MmTvPfee/nad8aMGZw7d85psXz++eeMHTvWacfLrz59+lxXxZHU1FRGjRpFo0aN6NKlC7GxsZdtk5KSQufOnWnbti0tW7bkpZdecmLE2TTB5IOIMKJjEMsn9ObGFtV4Y8lOhr4bRvSBU+4OTakizZMSTFH1ySefUKlSJWJiYhg/fjzPPPPMZdv4+/uzYsUKtm7dSkREBIsXL2bDhg1Oj0VvUy6AwHL+vPfXjiyOPsgLP21j6KwwxvRqwOM3NCbA19vd4SlVIC8v2Mb2xNNOPWaLWuV56ZaWV1w/ceJE9uzZQ7t27ejXrx/VqlXju+++IzU1lWHDhvHyyy9z9uxZRo4cSUJCApmZmbzwwgscPnyYxMRE/vKXv1C1alVWrlx52bEzMzN54IEHCA8PR0S4//77GT9+PH369OHNN98kJCSEo0ePEhIScvF//fHx8QwcOJB9+/Zx11138dJLL+X6/aNGjeKVV15hwYIFnD9/nu7du/PBBx8gIvTp04f27duzefNmkpKS+PLLL5kyZQpRUVGMGjWKV199ldjYWAYOHEiXLl3YsmULTZo04csvv6R06dKXjGHp0qW89NJLpKam0rBhQz777DPKli17yTY//fQTkyZNAuC2225j7NixGGMuuWAvIhf3S09PJz093SU3d+gMxgkGtqrJ8vG9Gd6+Nv/5bQ+DZ65hU+xxd4elVJEzdepUGjZsSEREBP369WP37t1s3LiRiIgINm/ezOrVq1m8eDG1atVi69atREdHM3DgQMaNG0etWrVYuXJlrskFICIiggMHDhAdHU1UVBT33XffNePZuHEjX3/9NREREfzvf/8jPDw81+8HGDt2LJs2bSI6Oprz58+zcOHCi8fx8/Nj9erVPPzwwwwdOpRZs2YRHR3N559/zrFjxwDYuXMnY8aMITIykvLly182kzt69Civvvoqy5cv548//iAkJITp06cD8OKLLzJ//nwADhw4QJ06dQDw8fGhQoUKF7/DUWZmJu3ataNatWr069ePLl26XPP343rpDMZJKpT25Y3b2zKkXS2e/T6K299fz9+61ePpgc0o66+/zaroudpMozAsXbqUpUuX0r59ewCSk5PZvXs3PXv25Mknn+SZZ57h5ptvpmfPnnk6XoMGDdi7dy+PPfYYN910E/3797/mPv369aNKlSoADB8+nLVr1zJ48OBcv3/lypVMmzaNc+fOcfz4cVq2bMktt9wCwJAhQwBo3bo1LVu2pGbNmhdjio+Pp2LFitSpU4fQ0FAA7r77bt5++22efPLJi7Fs2LCB7du3X9wmLS2Nbt26AfDKK69c3M6Yy+9uzW124u3tTUREBCdPnmTYsGFER0fTqlWra/6eXA+dwThZz8aBLHmiF/eFBjN7QxwD3lrNbzuPuDsspYocYwzPPvssERERREREEBMTwwMPPECTJk3YvHkzrVu35tlnn73kh+vVVKpUia1bt9KnTx9mzZrFgw8+CFj/y8/Ksm7SyflQYc4fzCKS6/enpKTw6KOPMnfuXKKionjooYcuOZa/vz8AXl5eF99f+JyRkXHF78r5+9GvX7+Lvx/bt2/nk08+uWycQUFBxMfHA5CRkcGpU6eoXLnyFX9fKlasSJ8+fVi8ePEVt8kvlyYYERkoIjtFJEZEJuayfrSIJIlIhP160GHdNBHZJiI7RORtsX+3RWSUiETa66Y5bO8vIt/a3/W7iAS7cmxXU8bfh5duacnch7tTys+b0Z9tYsJ3EZw4m+aukJQqEsqVK8eZM2cAGDBgAJ9++inJydajAAcOHODIkSMkJiZSunRp7r77bp588kn++OOPy/bNzdGjR8nKymLEiBFMnjz54n7BwcFs3rwZgLlz516yz7Jlyzh+/Djnz5/nxx9/JDQ0NNfvv5BMqlatSnJy8mXHyYv9+/ezfv16AL755ht69OhxyfquXbsSFhZGTEwMAOfOnWPXrl2XHWfIkCF88cUXF8fTt2/fy5JVUlISJ0+eBOD8+fMsX76cZs2aXXfM1+Kyczci4g3MAvoBCcAmEZlvjNmeY9NvjTFjc+zbHQgF2tiL1gK9RSQKeAPoaIxJEpEvROQGY8yvwAPACWNMIxG5A3gdGOWq8eVFx3qV+HlcD95dEcN/ftvD6l1JvDykFYNb19CnpZXKRZUqVQgNDaVVq1YMGjSIu+666+JpoLJly/LVV18RExPDU089hZeXF76+vvznP/8BYMyYMQwaNIiaNWvmeh3mwIED3HfffRdnK1OmTAHgySefZOTIkcyePZu+fftesk+PHj245557iImJ4a677iIkJIQlS5Zc9v0VK1bkoYceonXr1gQHB9OpU6frHnvz5s354osv+Pvf/07jxo155JFHLlkfGBjI559/zp133klqaioAr776Kk2aNOHFF18kJCSEIUOG8MADD3DPPffQqFEjKleuzJw5cwBITEzkwQcf5JdffuHgwYPce++9ZGZmkpWVxciRI7n55puvO+ZrkdzO1znlwCLdgEnGmAH252cBjDFTHLYZDYTkkmC6Ae8CPQABVgP3AGWBKcaYG+3t7gG6GWMeFZEl9vetFxEf4BAQaK4ywJCQEFNYHS23J57mmXmRRB04Rf8W1Xn11lZUKx9QKN+tVF7t2LGD5s2buzuMEic2Npabb76Z6Ohod4dymdz+TojIZmNMyLX2deUpstpAvMPnBHtZTiPsU15zRaQOgDFmPbASOGi/lhhjdgAxQDMRCbaTyK1AnZzfZ4zJAE4BVZw/rPxpUas8PzzanYmDmrFqVxI3TF/Fd5vic70gp5RSxYErE0xu54By/jRdAAQbY9oAy4EvAESkEdAcCMJKHH1FpJcx5gTwCPAtsAaIBTKu4/sQkTEiEi4i4UlJSdc9qILw8fbi4d4NWfR4T5rXKM/T8yK555ONxB/Xh8OUcqYuXbrQrl27S15RUVHuDuuKgoODPXL2UlCuvH82gezZBVjJItFxA2OM483ZH2FdNwEYBmwwxiQDiMgioCuw2hizACsxISJjgMwc35dgz24qAJc9jGKM+RD4EKxTZAUYX741CCzLnDFd+Xrjfl5f9Cf931rNUwOacm/3YLy99NqMUgX1+++/uzsEhWtnMJuAxiJSX0T8gDuA+Y4biEhNh49DgB32+/1YF/V9RMQX6H1hnYhUs3+tBDwKfGzvMx+4135/G7Diatdf3M3LS7inaz2Wju9FlwaVeWXhdm57fx27D1/5LhillCpKXJZg7OsgY4ElWMnhO2PMNhF5RUSG2JuNs2833gqMA0bby+cCe4AoYCuw1Z65AMwUke1AGDDVGHPhPr1PgCoiEgNMAC67LdoT1apYis9Gd2LGqHbEHj3LTW+v5e1fd5OWocUzlVJFm8vuIisKCvMusrw4mpzKpPnbWBh5kGY1yjHttja0Caro7rBUCaJ3kamcPPUuMnWdqpb15927OvDhPR05fjaNW2eFMeWXHaSkZ157Z6WU8jCaYDxQ/5Y1WDahN6M61eGD1XsZOGM1G/ZeXqxOqeLGk8r1F+d+MBdkZmbSvn17lzxkCZpgPFaFUr5MGd6G/z7YhSwDd3y4ged/iOJMSrq7Q1PKZTwpwRRVeekHc8HMmTNdekpUy/x6uO6NqrL4iZ5MX7qLT8P2seLPI7w2rBV9m1V3d2iquFs0EQ45+dmRGq1h0NQrrtZ+MIXTDwYgISGBn3/+meeff/5i2X9n0xlMEVDaz4d/3dyCeY90p6y/D/d/Hs4Tc7ZwXItnqmJG+8EUXj+YJ554gmnTpuHl5bo0oDOYIqR93UosHNeD91buYdbKGFbvPsqkIS25pU1NLZ6pnO8qM43CoP1gXNcPZuHChVSrVo2OHTvy22+/Xfs3L590BlPE+Pt4M75fExaO60FQpVKM+2YLD325mUOnUq69s1JFiPaDcV0/mLCwMObPn09wcDB33HEHK1as4O67777G7+D10wRTRDWrUZ7vH+nO84ObszYmiX7TV/HNxv1aPFMVadoPpnD6wUyZMoWEhARiY2OZM2cOffv25auvvrrumK9FT5EVYT7eXjzUqwH9WlRn4veRPPt9FPMjEpk6ojX1qpRxd3hKXTftB1M4/WAKiz7J70FP8hdEVpZhzqZ4pvyyg/SsLJ7s35T7Qutr8Ux1XfRJfvfQfjDKo3l5CXd1qcvSCb0IbViVV3/ewfD/rGPnIS2eqZRyD00wxUzNCqX4+N4QZt7Rjvjj57j5nTXMWL5Li2eqEkX7wXgGvQZTDIkIQ9vVpkejqryycDszlu9mUdQhXr+tDe3qaPFMdXW5PZRX1Gg/GOco6CUUncEUY1XK+jPzjvZ8cm8Ip86nM/y9MF77eTvn07R4pspdQEAAx44d07sRFcYYjh07RkBAQL6PoTOYEuCG5tXpVL8yUxf9yUdr9rFk22GmjmhN94ZV3R2a8jBBQUEkJCRQ2O3ElWcKCAggKCgo3/vrXWTF5C6yvFq/5xgTv48k7tg57uxch2cHN6d8gK+7w1JKFSF6F5nKVbeGVVj8eC/G9GrAt5vi6Td9Fcu3H3Z3WEqpYkgTTAlUys+b5wY354dHQ6lU2o8HvwznsW+2cCw51d2hKaWKEU0wJVjbOhWZP7YHE/o1YXH0QW6cvoqfIg7oBV6llFNoginh/Hy8GHdDY34e15N6Vcrw+JwIHvginMST590dmlKqiNMEowBoUr0c8x7pzgs3t2D9nmP0f2s1X22IIytLZzNKqfzRBKMu8vYSHuhRnyVP9KJtnQr868do7vxoA/uOnnV3aEqpIkgTTH4U82sUdauU5qsHuvD6iNZsP3iagTNW88GqPWRkarkZpVTeaYLJjz++hLc7wPzHYOu3cCrB3RE5nYgwqlNdlk/oTa8mgUxZ9CfD/7OOHQdPuzs0pVQRoQ9a5udBy93LYNMnELcOUk9ZyyrWg+CeEBwK9UKhUj3nButGxhh+jjrISz9t49T5dB7t05B/9G2Ev4+3u0NTSrlBXh+01ARTkCf5szLhcDTEhkGc/Tp/wlpXoY6VaIJDIbgHVKoPRbyA4ImzaUxeuJ3vtxygcbWyvH5bGzrUreTusJRShUwTTB44vVRMVhYk7bASTuwaa4Zz7qi1rlyt7NlNcA+o0qjIJpyVO4/w/PdRHDydwn3d6/PkgCaU9tOydkqVFJpg8sDltciMgaSdELc2e5aTbJdlKVsd6nW3E05PCGxapBLOmZR0pi3eyewNcQRVKsXU4W3o0ViLZypVEmiCyYNCL3ZpDBzbY89uwqykcybRWle6qpVwgntYSadaC/Dy/Hswft97jInfR7Hv6FlGhgTx/E0tqFBKi2cqVZxpgskDt1dTNgZO7Mue3cSuhVPx1rpSlaxEc+E6TvVW4OWZF9VT0jOZsXw3H63ZS5Uyfky+tRUDWtZwd1hKKRfRBJMHbk8wuTkRlz27iVsLJ2Kt5f4VoF637Gs4NdqAt2dd94hKOMXT8yLZcfA0N7WuyaQhLQks5+/usJRSTuYRCUZEBgIzAW/gY2PM1BzrRwNvAAfsRe8aYz62100DbsJ6VmcZ8LgxxojIncBzgAESgbuNMUdFZBLwEHChU9JzxphfrhafRyaYnE4lWDcLxK6xks7xPdZyv3JQt6t940APqNUOvN1/aio9M4sPV+9l5vLdlPb35sWbWzCsfe0i34JXKZXN7QlGRLyBXUA/IAHYBNxpjNnusM1oIMQYMzbHvt2xEk8ve9Fa4Fn710SghZ1UpgHnjDGT7ASTbIx5M68xFokEk9Ppg9m3RMeGwdGd1nLfMlC3S/YMp1YH8PFzW5gxR87w9NxI/th/kt5NAvn38NbUrljKbfEopZwnrwnGledYOgMxxpi9dkBzgKHA9qvuZTFAAOAHCOALHLbfC1BGRI4B5YEY54fuwcrXhNa3WS+A5CMOp9TCYMVka7lPKajTyZrdBIdC7RDwzX9v7evVqFo5/vdwd2avj2Xakp30n76KZwY14+4u9fDy0tmMUiWBKxNMbSDe4XMC0CWX7UaISC+s2c54Y0y8MWa9iKwEDmIllHeNMTsAROQRIAo4C+wG/uFwrLEi8jcgHPinMeaEswflccpWg5bDrBfA2WOwf511w0BsGPw2BTDg7Q9BnbKfxanTGXxdO6Pw9hJGh9bnhubVee6HKF78aRsLtiYydUQbGgaWdel3K6Xcz5WnyG4HBhhjHrQ/3wN0NsY85rBNFazTWqki8jAw0hjTV0QaYV27GWVvugx4BlgPLAbGAHuBd4BDxphXRaQ6cBRr9jMZqGmMuT+XuMbY+1O3bt2OcXFxLhi9Bzl/AuLWZ9+ldigSTBZ4+ULtjtbptOBQqNMF/Mq4LAxjDHM3JzB54XZSMrJ44sbGjOnZAB9vz78VWyl1KU+4BtMNmGSMGWB/fhbAGDPlCtt7A8eNMRVE5CkgwBgz2V73IpACrASmGmNusJf3AiYaYwbnOFYwsNAY0+pqMRbJazAFlXIK9m+wkk1cGCRGgMkELx+o1T77Gk6dLhBQ3ulff+RMCi/+uI3F2w7RqnZ5Xh/Rhpa1Kjj9e5RSruMJCcYH67TXDVh3iW0C7jLGbHPYpqYx5qD9fhjwjDGmq4iMwrojbCDWKbLFwAxgs/1qY4xJEpHJQGljzD9zHGs80MUYc8fVYiyRCSan1DMQ/3v2NZwDmyErA8QLarbLvkutblcoVdFpX7so6iAv/LSNE+fSeLh3Ax7r25gAX898zkcpdSm3Jxg7iMFYicEb+NQY85qIvAKEG2Pmi8gUYAiQARwHHjHG/GnPZt7DuovMAIuNMRPsYz4MPA6kA3HAaGPMMRGZDbSzt48F/n4h4VyJJphcpJ2F+I3ZNw4cCIfMNECgRmv7lFoPqNsNSlcu0FedPJfG5IU7mPdHAg0CyzBtRBtCggt2TKWU63lEgvF0mmDyIP08JGzKnuEkbIKMFECgesvsSgP1QqFM/mqRrdqVxHPfR5F46jz3dgvmqQFNKePvWQ+RKqWyaYLJA00w+ZCRap1Gu1BpYP/vkHHeWhfYLLuWWnAP6w63PEpOzeDNJTv5Yn0stSqUYsrw1vRqEuiaMSilCkQTTB5ognGCjDRI3JJdMTr+d0hLttZVaZx9DSc4FMrXuubhwmOP8/S8SPYmneW2jkH866bmVCztvgdGlVKX0wSTB5pgXCAzAw5uza4YvX8DpNptlis3yJ7d1AuFinVyPURKeibvrNjN+6v2Uqm0H5OHtmRQ65qFOAil1NVogskDTTCFICvTevbGsetnyoU203Xt2Y09w6lY75KeONsST/H03Ei2JZ5mUKsavDy0JdXKFV41AqVU7jTB5IEmGDfIyoTD27If/IxbB+ePW+vKB13a9bNyA9KzDB+t2cuM5bsJ8PHihZtbcFvHIC2eqZQbaYLJA00wHiArC5L+zE44sWsd2kzXvHiX2v7yHfjnr2fZFHeSno2r8u9hralTubR7Y1eqhNIEkweaYDyQMXB0V3algdgwSD5krSpTjbiy7Zh9MIhNtGB4/xv4W/f6WjxTqUKmCSYPNMEUAcbA8b3Z/XDiwuC01T7omCnH7oA2NOzUn8BWN0C1lkWizbRSRZ0mmDzQBFMEGQMnYjGxa9m/ZRm+8euoZfeYMwEVEccHP2u09tg200oVZZ7QD0Yp5xOByvWRyvWp1+Eeks6k8sK8Xzm7axUD/PbQ52A0/jt/trb1r5Dd9TO4B9Ro63FtppUqzvRfmyrSAsv5M3n0YBZHd+BfP0Vz/GgaEzqX4aF6h/CLt0+p7V5ibexXLkfXz/Ye0WZaqeJKT5HpKbJi49S5dP79yw6+DY+nQdUyTB3Rhs71K8OZQ5d2/Uz609rBt7TVeC24h/U8Tu0O4OPv3kEoVQToNZg80ARTPK3dfZSJ30eScOI893StxzODmlHWsXhmclL2Q5+xYXDE7iDhE2B3/bQrDQR1KtQ200oVFZpg8kATTPF1Li2DN5fs4rN1+6hZPoDXhrfmL02vUHzz3HHrgc+4MOtutUPRZLeZDsmuGB3UGfz02RulNMHkgSaY4m9z3AmemRdJzJFkhrevzQs3t6BSmWsUzzx/4tKunwe3OrSZ7nBp10//soUzEKU8iCaYPNAEUzKkZmQya0UM7/22h4qlfXl5SCsGt66R93IzKaftrp/2sziJW6w20+Jt3Sjg2PXTBW2mlfI0mmDyQBNMybI98TTPzIsk6sAp+reozuRbW1G9fD6usaQmWwnnYtfPzZCVbreZbps9w6nbzaltppXyFJpg8kATTMmTkZnFJ2v3MX3ZLvx8vPjXTc0ZGVKnYMUz085BwsZLu35ebDPdKrsfTr3QAreZVsoTaILJA00wJde+o2d5Zl4kG/cdJ7RRFaYMa0PdKk66gJ+eAgfCs4t3XmwzjVXO5kKyqRcKZbVrpyp6NMHkgSaYki0ry/DfjfuZuuhPMrMMTw5oyujuwXg7u3hmRioc+OPSrp/p56x1VZtm98Op1wPKVXfudyvlAppg8kATjAJIPHme53+IYuXOJNrXrci0EW1oXL2c674wM926UeDCXWr7Nzi0mW50adfPCrVdF4dS+aQJJg80wagLjDH8FJHIywu2kZyawWN9G/Nw74b4+RRCdebMDDi01aHr57rsNtOV6mfPboJDrS6gSrmZJpg80ASjcjqanMrLC7azYGsizWqU4/URbWhbp5DvBMvKhENRl5a3STlpratQ1+GUWihUCr6kzbRShcFpCUZEGgIJxphUEekDtAG+NMacdEqkbqQJRl3Jsu2H+dePUSSdSeWhng144sYmlPJzU+n/rCyrnE1smHUdJ24dnDtmrStfO7vSQL0eUKWhJhzlcs5MMBFACBAMLAHmA02NMYOdEKdbaYJRV3M6JZ0pv+zgm43xBFcpzdQRbejaoIq7w7ISztGdDl0/18JZqycOZWtkz26Ce0DVJppwlNM5M8H8YYzpICJPASnGmHdEZIsxpr2zgnUXTTAqL9bFHGXi91HsP36Ov3apy8RBzSgX4EFl/o2Bo7uz71KLC4MzB611ZQKhXnf7Gk4PCGymXT9VgTkzwfwOzACeB24xxuwTkWhjTCvnhOo+mmBUXp1Py+T/lu7k07B9VC8fwGvDWtG3mYfeUnyxzfTa7Os4pxOsdaUqWwnnwl1q1VtpwlHXzZkJpgXwMLDeGPONiNQHRhljpjonVPfRBKOu15b9VvHMXYeTGdquFi/e3IIqZT28h4wxcDIue3YTu9b6DBBQAep2z75xoEYbbTOtrskld5GJSCWgjjEmsiDBeQpNMCo/0jKyeO+3GGatjKFcgC+ThrTkljY1C1ZuprCdjM9ONnFh1owHwL+8VbSzXigE97Rqq2mbaZWDM2cwvwFDsNorRwBJwCpjzAQnxOlWmmBUQew8dIan50WyNf4kNzavxqu3tqZGhSLaoOx0onV32oWK0cd2W8v9ylptCS7cpVarPfhco92BKvacmWC2GGPai8iDWLOXl0Qk0hjTxlnBuosmGFVQmVmGz8L28ebSnfh6efHcTc25o1MBi2d6gjOHL+36mbTDWu5b2u762dNKOrU7apvpEsiZCSYK6A98ATxvjNmU1wQjIgOBmYA38HHO6zYiMhp4AzhgL3rXGPOxvW4acBPgBSwDHjfGGBG5E3gOMEAicLcx5qiIVAa+xbqdOhYYaYw5cbX4NMEoZ4k7dpaJ86JYv/cY3RpUYeqI1tSrUsbdYTnP2aOXPvh5ONpafqHN9MWun53At5R7Y1Uu58wEczvwAhBmjHlERBoAbxhjRlxjP29gF9APSAA2AXcaY7Y7bDMaCDHGjM2xb3esxNPLXrQWeNb+NRFoYSeVacA5Y8wk+/1xY8xUEZkIVDLGPHO1GDXBKGcyxjBnUzz//nkH6VlZ/LNfU+7vUd/5xTM9wbnjsH+9lXBi11iVBzDg7Qe1Q7KfxanTGfyKUaJVQN4TzDWv3hlj/gf8z+HzXuCqycXWGYixt0dE5gBDge1X3cv+GiAA8AME8AUO2+8FKCMix4DyQIy9z1Cgj/3+C+A34KoJRilnEhHu7FyXvzStxr9+jOK1X3awMDKRabe1pWkNFxbPdIfSlaHZTdYL4PxJq2jnhWdx1kyH1W+Alw/U6mAlnIttpovZ74W6omsmGBEJAt4BQrF+8K/FOl2VcI1dawPxDp8TgC65bDdCRHphzXbGG2PijTHrRWQlcBArobxrjNlhx/MIEAWcBXYD/7CPU90YcxDAGHNQRKpda2xKuUKNCgF89LcQFkQeZNL8bdz8zhoe7dOIf/ylUeEUz3SHUhWh6UDrBZB6BvbbbabjwmDdO7D2LbvNdDuHrp9drVulVbGUl1Nky4D/ArPtRXcDfzXG9LvGfrcDA4wxD9qf7wE6G2Mec9imCpBs1zl7GOu6SV8RaYR17WaUveks/FdnAAAeo0lEQVQyrNnIemAxMAbYi5X4DhljXhWRk8aYig7HPmGMqZRLXGPs/albt27HuLi4q45fqYI4fjaNVxZs48eIRJpUL8u029rSrrCLZ3qCtLNWH5yLXT/Ds9tM12ht3TRQLxTqdYNSl/2zVR7GqbXIjDHtrrUsl/26AZOMMQPsz88CGGOmXGF7b6xrKBXssjQBxpjJ9roXgRRgJTDVGHODvbwXMNEYM1hEdgJ97NlLTeA3Y0zTq8Wo12BUYVnx52Ge/yGaw6dTuD+0Pv/s39R9xTM9Qdo5q9PnhRsHEjZBZiogVnUBx66fZTyg/pu6hNOuwQBHReRu4Bv7853AsTzstwlobD/5fwC4A7grR5A1L5zWwnrWxr4Xkv3AQyIyBesUWW+scjUHgBYiEmiMScK6geDCPvOBe4Gp9q8/5SFGpQpF32bVWTq+MlMX/cnHa/exdPthpg5vTfdGVd0dmnv4lYYGva0X2G2mN9sJZw1s/gJ+f99aV62FQ8XoUCirZ7+LirzMYOoC7wLdsK7BrAPGGWP2X/PgIoOxEoM38Kkx5jUReQUIN8bMtxPIECADOA48Yoz5057NvId1F5kBFl94sNM+lfY4kA7EAaONMcfs023fAXWxEtTtxpjjV4tPZzDKHTbsPcbEeZHEHjvHHZ3q8Ozg5lQo5UHFMz1BRhok/uHQ9fN3SD9rravaJLuWWnAPKFfDvbGWQC5tOCYiTxhjZuQrMg+iCUa5y/m0TGYs38VHa/YSWM6fV29tTb8WHlo80xNkpsPBrdmVBvZvgLQz1rrKDS/t+lkhyL2xlgCuTjD7jTFFvnerJhjlbpEJJ3l6biR/HjrDzW1qMmlIS6p6evFMT5CZAYciHR7+XAepp6x1lYKzk029UKhUz62hFkeuTjDxxpg6+YrMg2iCUZ4gLSOLD1bt4Z0VMZTx9+alW1oytF2tol9upjBlZVrVBS7cpRYXBuftQh4V6mRfwwnuAZXqaxO2AtIZTB5oglGeZPdhq3jmlv0n6dusGq/e2opaFbXsSr5kZVn102LX2tdx1sG5o9a6crUu7fpZpZEmnOtU4AQjImewLrBftgooZYwp8jW8NcEoT5OZZfhiXSxvLNmJt5cwcVAz7upcF6/iWG6mMBkDSTuzKw3EroWzR6x1Zas73KXWAwKbasK5BpfOYIoLTTDKU+0/do5nf4gkLOYYnetX5vURbahfVWt6OY0xcCzm0q6fZxKtdaWrXtr1s1oL7fqZgyaYPNAEozyZMYb/hScw+eftpGVkMb5fEx7sUR8fb/1h53TGwIl92bObuDA4ZVe6KlUp+6HP4AttpkvwQ7JogskTTTCqKDh8OoUXfoxm6fbDtK5dgddHtKFFrfLuDqv4OxHncJfaWjgRay0PqAB1u2XPcGq0KXFdPzXB5IEmGFVUGGP4JeoQL82P5uS5dB7p05CxfRvh71Oy/yddqE4lZCeb2DA4vsda7lfOKtp5setnO/Au3g/OaoLJA00wqqg5cTaNyT9v5/s/DtCoWlleH9GGjvW0OKRbnD54adfPozut5b5loG6X7LvUanUodm2mnVnsMre7yU4B4cA/L/R7KYo0waiiauXOIzz/fRQHT6cwunswT/ZvShn/knWaxuMkH7m06+cRu/WVTymo08l++LOH1WbaN8C9sRaQMxPMy1hdJP+LdYvyHUANYCdW7bA+BY7WTTTBqKIsOTWDaYv/5Mv1cQRVKsWU4a3p2TjQ3WGpC84eu3SGczgaq+unv9Va2rHrZxFrM+3MBPO7MaZLjmUbjDFdRWSrMaZtAWN1G00wqjjYuO84E+dFsvfoWUaGBPH84BZUKF28rwEUSedPQNx6O+GstUrdmCzw8rVmNcF2eZs6XTy+zbQzE8x64C1grr3oNmCCnWCu2RfGk2mCUcVFSnomM3/dzYer91K5jB+Th7ZiYCutMuzRUk5ZRTsv3BadGAEm024z3d6+htPTup7jYW2mnZlgGmB1l+xmL1oPjMfqzdLRGLO2gLG6jSYYVdxEHzjF03Mj2X7wNDe1topnBpbT4plFQuoZu+unfZda4h+QlWG1ma7ZNvsutbpdrRbVbqR3keWBJhhVHKVnZvHh6r3MXL6bUn7evHhzC4Z3qK3FM4uatLMQvzH7Gs6BcMhMAyS7zXRwqPVMTunKhRqaM2cwQcA7QCjW3WRrgceNMQnOCNSdNMGo4izmSDLPzItkc9wJejcJ5LVhrQiqVNrdYan8Sj9vtZa+cJda/EaHNtMtL+36Wca1nVKdmWCWYd1BNttedDfwV2NMvwJH6WaaYFRxl5VlmL0hjtcX/4kAzwxqxt1d6mnxzOIgI9VqMx1rt5mO3wgZ5611gc0vrRjt5DbTzkwwl13IL+oX9y/QBKNKivjj53juhyjW7D5Kp+BKTB3RhoaBZd0dlnKmjDRI3JJdaSD+d0hLttZVaWz3w+lpJZ3yNQv0Vc5MMMuBz4Fv7EV3AvcZY24oUIQeQBOMKkmMMcz74wCTF27nfHomT9zYmId6NsBXi2cWT5kZ2W2m4+w206mnrXWVG8ANL0LLYfk6tDMTTF3gXay7yAywDhhnjNmfr8g8iCYYVRIdOZPCSz9tY1H0IVrWKs/rI9rQqnYFd4elXC0r03r25sI1nM5joOFf8nUoV3e0fMIYMyNfkXkQTTCqJFsUdZAXftrGiXNpPNy7AY/1bUyArxbPVNeW1wST37nxhHzup5TyEINa12T5hF4Ma1+bWSv3MPjtNYTHHnd3WKoYyW+C0VtQlCoGKpb2483b2/Ll/Z1JTc/i9g/W89JP0SSnZrg7NFUM5DfBlNynM5Uqhno1CWTp+F7c2y2YLzfEMeCt1azaleTusFQRd8UEIyJnROR0Lq8zQK1CjFEpVQjK+PswaUhL/vf3bgT4enHvpxv553dbOXkuzd2hqSLqignGGFPOGFM+l1c5Y4w2nlCqmAoJrszP43oy9i+N+DHiADdOX82iqIPuDksVQXoDvFLqMgG+3jw5oCnzx4ZSvbw/j3z9Bw/P3syR0ynuDk0VIZpglFJX1LJWBX76RyjPDGzGip1HuHH6Kr4Lj6ckF8lVeacJRil1VT7eXjzSpyGLH+9JsxrleXpuJH/7dCPxx8+5OzTl4TTBKKXypEFgWeaM6crkoS35I+4EA2as5rOwfWRm6WxG5U4TjFIqz7y8hHu6BbN0Qm86BVfm5QXbGfnBemKOnHF3aMoDuTTBiMhAEdkpIjEiMjGX9aNFJElEIuzXgw7rponINhHZISJvi6Wcw7YRInJURGZc61hKKeeqXbEUn9/Xiekj27InKZnBM9fy7ordpGdmuTs05UFcdruxiHgDs4B+QAKwSUTmG2O259j0W2PM2Bz7dsdqcNbGXrQW6G2M+Q1o57DdZuD7qx1LKeUaIsLwDkH0bBzIpAXbeHPpLn6OOsS0EW1oHaTFM5VrZzCdgRhjzF5jTBowBxiax30NEAD4Af6AL3DYcQMRaQxUA9Y4LWKl1HULLOfPrLs68ME9HTmWnMqt74UxddGfpKRnujs05WauTDC1gXiHzwn2spxGiEikiMwVkToAxpj1wErgoP1aYozZkWO/O7FmLOZqx8pJRMaISLiIhCclaSkMpZxlQMsaLJvQm9s6BPH+qj0MmrmG3/cec3dYyo1cmWByK4iZ83aTBUCwMaYNsBz4AkBEGgHNgSCspNRXRHrl2PcOspugXfFYlwVgzIfGmBBjTEhgYOB1DkkpdTUVSvny+m1t+PrBLmRkZTHqww288GM0Z1LS3R2acgNXJpgEwHEWEQQkOm5gjDlmjEm1P34EdLTfDwM2GGOSjTHJwCKg64X9RKQt4GOM2ZyHYymlClloo6oseaIX94fW56vfreKZK3cecXdYqpC5MsFsAhqLSH0R8cOaccx33EBEHBtDDwEunAbbD/QWER8R8QV6O6wD6/SY4+zlasdSSrlBaT8fXrylBfMe6U4Zfx/u+2wTE76N4MRZLZ5ZUrjsLjJjTIaIjAWWAN7Ap8aYbSLyChBujJkPjBORIUAGcBwYbe8+F+gLRGGdVltsjFngcPiRwOAcX3mlYyml3KhD3UosHNeDWSv38N7KGFbtSuLloS25qXVNRLS1VHGWr5bJxYW2TFaqcO04eJpn5kUSmXCKfi2q8+qtrahePsDdYanr5OqWyUopdd2a1yzP949057nBzVi9K4kbp6/i2037tXhmMaUJRilVqHy8vRjTqyFLnuhFi5rleWZeFH/9+Hf2H9PimcWNJhillFsEVy3DNw915bVhrYhMOMWAGav5ZK0WzyxONMEopdzGy0v4a5d6LJvQi24NqzB54XZG/Gcduw5r8cziQBOMUsrtalYoxSf3hjDzjnbEHTvLTW+vYeby3aRlaPHMokwTjFLKI4gIQ9vVZvmE3gxqVZO3lu9iyLtr2Rp/0t2hqXzSBKOU8ihVyvrz9p3t+fhvIZw8l86w98L49y87OJ+mxTOLGk0wSimPdGOL6iyd0ItRnery4eq9DJq5mvV7tHhmUaIJRinlscoH+DJleGv++1AXDHDnRxt47ocoTmvxzCJBE4xSyuN1b1iVxY/34qGe9ZmzcT/9p6/m1x2Hr72jcitNMEqpIqGUnzfP39SC7x8NpUIpXx74Ipxx32zhWHLqtXdWbqEJRilVpLSrU5EFj/Vg/I1NWBR9kH5vreaniANabsYDaYJRShU5fj5ePH5jYxY+1pM6lUvz+JwIHvwinIOnzrs7NOVAE4xSqshqWqMc3z/SnX/d1JywPUfpP301//19P1labsYjaIJRShVp3l7Cgz0bsOSJXrSqXYHnfojiro83EHv0rLtDK/E0wSilioV6Vcrw34e6MHV4a7YdOM3Amav5aPVeMjK13Iy7aIJRShUbIsIdneuybEJvejQK5LVfdjDiP+v489Bpd4dWImmCUUoVOzUqBPDR3zryzp3tSThxnpvfXsv0ZbtIzdByM4VJE4xSqlgSEW5pW4tlE3pzS9tavP3rbm55Zy1b9p9wd2glhiYYpVSxVrmMH2+Naseno0M4k5LB8P+sY/LC7ZxLy3B3aMWeJhilVInQt1l1lo7vxV+71OWTtfsYOGMN62KOujusYk0TjFKqxCgX4Murt7ZmzpiueHsJd338OxPnRXLqvBbPdAVNMEqpEqdrgyoserwnf+/dgO/C4+k3fRVLtx1yd1jFjiYYpVSJFODrzbODmvPjP0KpXMaPMbM3M/a/f3BUi2c6jSYYpVSJ1ibIKp75z35NWLrtMDdOX8UPWxK0eKYTaIJRSpV4vt5ePHZDY34e14P6Vcsw/tut3P/5JhJPavHMgtAEo5RStsbVyzH34e68dEsLNuw9Tv+3VjN7Q5wWz8wnTTBKKeXA20u4L7Q+S8f3ol2dirzwYzR3fLiBvUnJ7g6tyNEEo5RSuahTuTSzH+jMtNva8Oeh0wyauYb3V+3R4pnXQROMUkpdgYgwMqQOyyf0pk/TQKYu+pNb3wtje6IWz8wLlyYYERkoIjtFJEZEJuayfrSIJIlIhP160GHdNBHZJiI7RORtsZRz2DZCRI6KyAx7e38R+db+rt9FJNiVY1NKlRzVygfw/t0dee+vHTh0KoUh767l/5bu1OKZ1+CyBCMi3sAsYBDQArhTRFrksum3xph29utje9/uQCjQBmgFdAJ6G2POOGzbDogDvreP8wBwwhjTCHgLeN1VY1NKlTwiwuDWNVk2vjdD2tXinRUx3PT2WjbHafHMK3HlDKYzEGOM2WuMSQPmAEPzuK8BAgA/wB/wBQ47biAijYFqwBp70VDgC/v9XOAGEZECjUAppXKoVMaP6SPb8fl9nTiflslt76/j5QXbOJuqxTNzcmWCqQ3EO3xOsJflNEJEIkVkrojUATDGrAdWAgft1xJjzI4c+92JNfu5cP/gxe8zxmQAp4AqzhqMUko56tO0GkvG9+JvXevxWVgsA2asZs3uJHeH5VFcmWBymz3kvJl8ARBsjGkDLMeegYhII6A5EISVOPqKSK8c+94BfHOd34eIjBGRcBEJT0rSvwxKqfwr6+/Dy0Nb8b+Hu+Hn48U9n2zkqf9t5dQ5LZ4Jrk0wCUAdh89BQKLjBsaYY8aYC4V/PgI62u+HARuMMcnGmGRgEdD1wn4i0hbwMcZszu37RMQHqAAczxmUMeZDY0yIMSYkMDCwIONTSikAOgVX5pdxPXm0T0O+33KAG99axeJoLZ7pygSzCWgsIvVFxA9rxjHfcQMRqenwcQhw4TTYfqC3iPiIiC/Q22EdWKfHHGcv2Me+135/G7DCaDEhpVQhCfD15umBzfjpH6EElvXn4a828+jXmzlyJsXdobmNyxKMfR1kLLAEKzl8Z4zZJiKviMgQe7Nx9q3IW4FxwGh7+VxgDxAFbAW2GmMWOBx+JJcnmE+AKiISA0wALrstWimlXK1V7Qr8NDaUpwY0ZfmOI/Sbvpp5m0tm8UwpiYO+ICQkxISHh7s7DKVUMRVzJJmJ8yIJjztBryaB/HtYK4IqlXZ3WAUmIpuNMSHX2k6f5FdKKRdpVK0s3/29Gy8PaUl4rFU884t1sSWmeKYmGKWUciEvL+He7sEsHd+LkODKvDR/GyM/WM+eElA8UxOMUkoVgqBKpfnivk68eXtbdh9JZtDMNcxaGUN6MS6eqQlGKaUKiYhwW8cglk3oxY3Nq/HGkp3cOiuM6AOn3B2aS2iCUUqpQlatXADv/bUj79/dgSNnUhk6K4xpi/8kJb14Fc/UBKOUUm4ysFVNlo/vzfD2tXnvtz0MnrmGTbGXPR9eZGmCUUopN6pQ2pc3bm/L7Ac6k5aZxe3vr+fFn6JJLgbFMzXBKKWUB+jZOJAlT/RidPdgZm+IY8Bbq1m1q2jXS9QEo5RSHqKMvw+ThrRk7sPdCPD14t5PNzLhuwhOnktzd2j5oglGKaU8TMd6lfl5XE8e69uI+RGJ3Dh9Fb9EHXR3WNdNE4xSSnmgAF9v/tm/KfPH9qBmhVI8+vUf/H12OEdOF53imZpglFLKg7WoVZ4fHu3OxEHN+G1nEjdOX8V34fFFonimJhillPJwPt5ePNy7IYse70mzGuV5em4k93yykfjj59wd2lVpglFKqSKiQWBZ5ozpyuRbW7Fl/wn6v7Waz8L2kemhxTM1wSilVBHi5SXc07UeSyf0pkuDyry8YDu3v7+OmCNn3B3aZTTBKKVUEVS7Yik+G92Jt0a1Zd/RswyeuZZ3V+z2qOKZmmCUUqqIEhGGtQ9i2YTe9G9ZnTeX7uKWd9YSleAZxTM1wSilVBFXtaw/797VgQ/v6cjxs2kMnbWWKYt2uL14piYYpZQqJvq3rMGyCb0ZGVKHD1btZdDMNfy+95jb4tEEo5RSxUiFUr5MHdGGrx/sQkZWFqM+3MC/foziTEp6oceiCUYppYqh0EZVWfJELx7oUZ+vf9/PgLdWs/LPI4UagyYYpZQqpkr7+fDCzS2Y90h3yvj7cN/nmxj/bQTHzxZO8UxNMEopVcx1qFuJheN68PgNjVmwNZF+01ex4s/DLv9eTTBKKVUC+Pt4M75fExaO60FQ5dKU9fd1+Xf6uPwblFJKeYxmNcrz46PdERGXf5fOYJRSqoQpjOQCmmCUUkq5iCYYpZRSLqEJRimllEtoglFKKeUSmmCUUkq5hCYYpZRSLqEJRimllEuIMZ7Zy7kwiEgSEJfP3asCR50YTlGgYy4ZdMwlQ0HGXM8YE3itjUp0gikIEQk3xoS4O47CpGMuGXTMJUNhjFlPkSmllHIJTTBKKaVcQhNM/n3o7gDcQMdcMuiYSwaXj1mvwSillHIJncEopZRyCU0wuRCRgSKyU0RiRGRiLuvfEpEI+7VLRE46rLtXRHbbr3sLN/L8y++YRaSdiKwXkW0iEikiowo/+vwpyJ+zvb68iBwQkXcLL+r8K+Df67oislREdojIdhEJLszY86uAY55m/73eISJvS2HVuC+gPIy5roisFJEt9r/ZwQ7rnrX32ykiAwocjDFGXw4vwBvYAzQA/ICtQIurbP8Y8Kn9vjKw1/61kv2+krvH5OIxNwEa2+9rAQeBiu4ekyvH7LBsJvBf4F13j8fV4wV+A/rZ78sCpd09JleOGegOhNnH8AbWA33cPSZnjBnr2ssj9vsWQKzD+62AP1DfPo53QeLRGczlOgMxxpi9xpg0YA4w9Crb3wl8Y78fACwzxhw3xpwAlgEDXRqtc+R7zMaYXcaY3fb7ROAIcM0HsDxAQf6cEZGOQHVgqUujdJ58j1dEWgA+xphlAMaYZGPMOVcH7AQF+TM2QADWD2l/wBdwfRP7gsvLmA1Q3n5fAUi03w8F5hhjUo0x+4AY+3j5pgnmcrWBeIfPCfayy4hIPaxMv+J69/UwBRmz47rOWP8g97ggRmfL95hFxAv4P+ApF8foTAX5M24CnBSR7+3TKm+IiLdLo3WOfI/ZGLMeWIk1Iz8ILDHG7HBptM6RlzFPAu4WkQTgF6yZW173vS6aYC6X23nWK91qdwcw1xiTmY99PUlBxmwdQKQmMBu4zxiT5eT4XKEgY34U+MUYE3+F7T1RQcbrA/QEngQ6YZ1+Ge3sAF0g32MWkUZAcyAI64dsXxHp5ZIonSsvY74T+NwYEwQMBmbb/2ly+s8vTTCXSwDqOHwOInsKmdMdOJw2uc59PUlBxoyIlAd+Bv5ljNngkgidryBj7gaMFZFY4E3gbyIy1RVBOlFB/15vsU+7ZAA/Ah1cEqVzFWTMw4AN9unAZGAR0NUlUTpXXsb8APAdXJypBWDVJXP+zy93X5TytBfW/9b2Yk2XL1wka5nLdk2BWOxniexllYF9WBf4K9nvK7t7TC4esx/wK/CEu8dRWGPOsX40ReMif0H+jL3t7QPtz58B/3D3mFw85lHAcvsYvvbf8VvcPSZnjBkrWY623zfHSiICtOTSi/x70Yv8zmWs/6GNBZYAO4DvjDHbROQVERnisOmdWBfEjMO+x4HJwCb79Yq9zKMVZMzASKAXMNrhds92hRZ8PhVwzEVOAf9eZ2KdHvtVRKKwfhh9VHjR508B/4znYl1LjML6obvVGLOgkELPtzyO+Z/AQyKyFWvWNtpYtmHNbLYDi7H+E5F5+bfknT7Jr5RSyiV0BqOUUsolNMEopZRyCU0wSimlXEITjFJKKZfQBKOUUsolNMEopZRyCU0wSuVCRCaJyJNu+N7R+Sn/f73x2m0WBl97S6XyTxOMUiVTO6w6VJcREZ9CjkUVU5pgVIkiIn+zmyxtFZHZIlJPRH61l/0qInVz2Wec3WQrUkTm2Ms6i8g6u7rwOhFpai8fLSI/isgCEdknImNFZIK93QYRqWxv95uIzLD3jbYrUef83kARmScim+xX6DWG11ZEVojV7O4h+xizReRiuXYR+dp+ovsVYJRdeWGUPQP6UESWAl+KiLddNXmTPe6/OxzjKYflL1//n4IqMdxdO0df+iqsF1atpZ1AVftzZWABcK/9+X7gR/v9JOBJ+30i4G+/r2j/Wh6rRwrAjcA8+/1orD4a5bD64pwCHrbXvYVdsw2rgddH9vteQLTD/u/a7/8L9LDf1wV2XGVsk7BKmpTCKlwYj9UArrfDmCpg1cfzIUcNNXv/zUAp+/MYrOKlYNWmCseqT9Ufq2GVYP0HdSHQy91/tvryzJdOhVVJ0herJPtRsGrHiUg3YLi9fjYwLZf9IoGvReRHrErCYP2w/kJEGmOVNPd12H6lMeYMcEZETmElMbDqWrVx2O5C07bVYrVfrpjje28EWjh06i0vIuXsY+fmJ2PMeeC8iKwEOhtjfhSRWSJSzR7nPGNMhuTe/Xe+vT9YiaSNiNzmMN7G9vL+wBZ7eVl7+eorxKRKME0wqiQRrt3fIrf1N2HNMoYAL4hIS6yipiuNMcPE6k//m8P2qQ7vsxw+Z3Hpv7mc35XzsxfQzeGH/rVc6Xizgb9ilaS//yr7n3V4L8BjxpgljhvYfdqnGGM+yGNMqgTTazCqJPkVGCkiVQDs6yHrsH7wgvVDeK3jDnYjpjrGmJXA00BFrP+1VwAO2JuNzmc8o+zv6AGcMsacyrF+KVZl3AuxXKtK9VARCbDH1werojfA58ATAMaqmAtwBus03pUsAR4REV/7u5uISBl7+f0iUtZeXtueHSl1GZ3BqBLDWGXLXwNWiUgm1mmeccCnIvIUkATcl2M3b+ArEamA9b/6t4wxJ0VkGtYpsgnk0j46j06IyDqs6zm5zSzGAbNEJBLr3+pq4OGrHG8jVuO3usBkY0wigDHmsIjsIPv0HljtgCeKSAQwJZdjfQwEA3+IdT4tCbjVGLNURJoD6+3TbMnA3cCRvA1ZlSRarl8pNxCR37BuIggvhO8qjXX9p0MusySlXEZPkSlVjInIjcCfwDuaXFRh0xmMUkWIiNwHPJ5jcZgx5h/uiEepq9EEo5RSyiX0FJlSSimX0ASjlFLKJTTBKKWUcglNMEoppVxCE4xSSimX+H80w3bgGDXb6wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xf77e748>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch3_1.best_score_,gsearch3_1.best_params_))\n",
    "\n",
    "test_means = gsearch3_1.cv_results_['mean_test_score']\n",
    "test_stds = gsearch3_1.cv_results_['std_test_score']\n",
    "train_means = gsearch3_1.cv_results_['mean_train_score']\n",
    "train_stds = gsearch3_1.cv_results_['std_train_score']\n",
    "\n",
    "pd.DataFrame(gsearch3_1.cv_results_).to_csv('my_preds_subsample_colsample_3.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(subsample),len(colsample_bytree))\n",
    "train_scores = np.array(train_means).reshape(len(subsample),len(colsample_bytree))\n",
    "\n",
    "for i, value in enumerate(subsample):\n",
    "    pyplot.plot(colsample_bytree,-test_scores[i],label='test_subsample:'+str(value))\n",
    "    \n",
    "# for i, value in enumerate(colsample_bytree):\n",
    "#    pyplot.plot(subsample,train_scores[i],label='train_colsample_bytree:'+str(value))\n",
    "\n",
    "pyplot.legend()\n",
    "pyplot.xlabel('colsample_bytree')\n",
    "pyplot.ylabel('Log Loss')\n",
    "pyplot.savefig('subsample_vs_colsample_bytree1.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "两个参数subsample、colsample_bytree，横轴只能一个参数(X轴：colsample_bytree)，另外参数（subsample）用不同曲线表示。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Best: -0.586651 using {'subsample': 0.4, 'colsample_bytree': 0.7}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
